\documentclass{bioinfo}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}

\usepackage{hyperref}
\usepackage{fixltx2e}
\usepackage{graphicx,amsfonts,amsmath,amsthm,amssymb, siunitx, array}

\newcommand\blfootnote[1]{%
  \begingroup
  \renewcommand\thefootnote{}\footnote{#1}%
  \addtocounter{footnote}{-1}%
  \endgroup
}


\hypersetup{
    bookmarks=true,         % show bookmarks bar?
    unicode=true,          % non-Latin characters in Acrobats bookmarks
    pdftoolbar=true,        % show Acrobats toolbar?
    pdfmenubar=true,        % show Acrobats menu?
    pdffitwindow=false,     % window fit to page when opened
    pdfstartview={Fit},    % fits the width of the page to the window
    pdftitle={},    % title
    pdfauthor={},     % author
    %pdfsubject={Subject},   % subject of the document
    %pdfcreator={Creator},   % creator of the document
    %pdfproducer={Producer}, % producer of the document
    pdfkeywords={}, % list of keywords
    pdfnewwindow=true,      % links in new window
%     colorlinks=true,       % false: boxed links; true: colored links	%this is awesome, change as wanted
colorlinks=false,
    pdfborder={0 0 0},    %  hack!
    linkcolor=red,          % color of internal links
    citecolor=green,        % color of links to bibliography
    filecolor=magenta,      % color of file links
    urlcolor=cyan           % color of external links
}
\bibliographystyle{natbib}	%Has to be here (or somewhere else in the document at least)

\copyrightyear{}
\pubyear{}
\begin{document}
\firstpage{1}
\title{Prediction of transmembrane alpha helices}
\author{Michael Ederer, Alexander Giehl,  Kang-Hunn Lee,  Alice Meier,  Jonas Reeb, Stefan Stöckl}
\address{}
\history{}
\editor{}

\maketitle
% Use \begin{figure*} …\end{figure*} to get figure in double column.
% 
%       \citet{key}             ==>  Jones et al. (1990)  
%       \citep{key}            ==>  (Jones et al., 1990)
%	\citealp{key}		==> Jones et al., (1990) 	
%       \citep{key1,key2}  ==>  (Jones et al., 1990; Smith, 1989)
%       \citealt{key}          ==>  Jones et al. 1990 
%       \citeauthor{key}    ==>  Jones et al.
%       \citeyear{key}        ==>  1990


% \citealp{Bagos2010} show braces around year
% \cite{Bagos2010} dont show them



\begin{abstract}
\section{Motivation:}  %%%% 'DONE', review al:done
Integral membrane proteins are an abundant class of proteins, featuring specialised chemical properties that makes them intrinsically hard to handle in structure determination. At the same time
they play important roles in a multitude of cellular processed and contain a large group of known drug targets. Prediction of transmembrane helices in 1D allows for a first assessment of a protein's features
and can also give crucial help for structure determination.
\section{Results:} We introduce an SVM and jury-decision based transmembrane helix predictor, benchmarked with a full cross-validation on a set of 86 transmembrane proteins.
The method achieves a prediction performance of 71\% $\text{Q}_\text{ok}$  and 88\% for the $\text{Q}_2$.
\blfootnote{Abbreviations used: \textbf{ANN}, artificial neural network; \textbf{DBN}, dynamic Bayesian network; \textbf{EM}, electron microscopy; \textbf{HMM}, hidden Markov model; 
\textbf{IMP}, integral membrane protein; \textbf{MCC} Matthews correlation coefficient;
\textbf{NMR}, nuclear magnetic resonance; \textbf{TMH}, transmembrane helix; \textbf{SVM}, Support Vector Machine}
The discrimination between soluble and transmembrane proteins is achieved with a sensitivity of 98\% and a specificity of 87\%. 

\end{abstract}
\section{Introduction}  %%%% 'DONE', review al:done
Transmembrane proteins constitute around 20-30\% of the proteome \citep{Stevens2000a,Fagerberg2010a}. The large majority of these integral membrane proteins (IMPs) 
consists of proteins that cross the membrane with a segment assuming alpha-helical conformation.
The importance of this protein group is highlighted by their role as a drug targets. The large group of alpha-helical IMPs, GPCRs, make up around 30\% of current drug targets \citep{Jacoby2006a, Overington2006a}.
Although highly interesting, determination of the three-dimensional structure of IMPs is much harder than for their soluble counterparts. This is attributed to the special properties of transmembrane proteins. For structure determination they have to be extracted from the membrane, however the hydrophobic transmembrane regions then lie exposed to the water environment and would aggregate, destroying their native conformation.

The ensuing lack of training data for prediction methods is 
hard to tackle. Nonetheless prediction of transmembrane helices in 1D has been done early on, with one of the first methods being TopPred \citep{Claros1994}. These early methods did in fact not yet employ machine learning models but relied solely on the
strong signal that transmembrane segments are composed mostly of hydrophobic amino acids. With the increasing amount of high-resolution structures and low-resolution data about the position of helices determined by tagging experiments, machine learning
methods surfaced using all of the common models like HMMs \citep{Tusnady1998,Krogh2001,Kall2005}, ANNs \citep{Rost1996} or SVMs \citep{Yuan2004,Nugent2009b} and DBNs \citep{Reynolds2008}.

In this work we present a transmembrane helix prediction method based on two predictors, consisting of three SVMs each that are combined with a jury decision and predict the discrimination between soluble and transmembrane proteins, as well as the positions of transmembrane helices, respectively.


\input{methods.tex}

 %%%%%%%%%%%%%%%%% ---------------------------------------------------------------------------------------------------------------
\section{Results and Discussion}
The two predictors were evaluated separately, using the scores described above. Unfortunately, the jury decisions of the SolTMH and the Helix-predictor could not be validated because the small number of transmembrane proteins 
within each set leaves no room for an independent testing set. Therefore the performance of the jury is estimated by the average over the independent prediction performances of the jury predictors obtained during the cross-validation. 

It should be kept in mind that in this section set x denotes a training on this set and the according cross-training and testing set were chosen as described in \ref{tbl:setmatching}.

\subsection{Discrimination to soluble proteins}
The SolTMH-predictor was evaluated by MCC, sensitivity and specifictiy, according to \citealp{Nugent2009b}. The results are shown in Table~\ref{tbl:level1results}. The worst MCC lies at 0.78 with a 
specificity of 0.75, displaying a slight tendency to over prediction at protein level.

To further evaluate discrimination on protein level, a set of 100, so far unused, soluble proteins was randomly sampled from the remaining soluble data.
For this testing set the complete predictor with SolTMH and Helix level was applied.
6 of these 100 sequences were falsely assigned to be IMPS, resulting in an overall sensitivity of 0.94 on protein level. A look at the output revealed a pattern which matched 5 of the 6 proteins: 
In these, an approximately 10 residues long TMH was predicted at the beginning of the sequence starting around position 7. A check against annotation in UniProt confirmed that all 5 proteins contain a signal peptide. The 6th falsely assigned 
sequence had several helices predicted but never more than 2 adjacent to each other.

\begin{table}[!t]
\processtable{\textbf{Performance evaluation of SolTMH-predictor.} \textbf{Set} denotes the training set of the SVM, the according optimization and final test set are chosen as 
outlined in Table~\ref{tbl:setmatching}.\label{tbl:level1results}}
{
  \begin{tabular}{l|p{1.7cm}p{1.7cm}p{1.7cm}}
\toprule
\textbf{Set} & \textbf{Sensitivity} & \textbf{Specificity}& \textbf{MCC}\\
\midrule
1 & 0.96 & 0.96& 0.93\\
2 & 1 & 0.75& 0.78\\
3 & 1 & 0.90& 0.90\\
\midrule
Avg & 98 & 87& 87\\
\botrule
\end{tabular}}{}
\end{table}
\subsection{Prediction of transmembrane helices}
The  evaluation of the Helix-predictor, denoting the per-residue as well as the per-segment performance is shown in Table~\ref{tbl:level2results}. 
The accuracy on residue level is 88\% but as IMPs generally have a greater abundance of non-transmembrane residues (in the 3 sets the proportion of TMH to non-TMH residues was 30:70), this measure is strongly biased towards non-TMH residues and of 
limited use.
A possible consequence of this bias is reflected by the $\text{Q}_{\text{2T}}^{}$ scores. Here, a higher $\text{Q}_{\text{2T}}^{\text{\%obs}}$ than $\text{Q}_{\text{2T}}^{\text{\%pred}}$ depicts a slight tendency towards under-prediction.
$\text{Q}_{\text{2N}}^{}$ scores consistently remain higher due to the uneven class distribution in the underlying data.

With the average $\text{Q}_{\text{htm}}^{\text{\%pred}}$ of 95\% and $\text{Q}_{\text{htm}}^{\text{\%obs}}$ of 93\% a satisfying performance on segment level is displayed. 
Nonetheless, the most strict measure, Q\textsubscript{ok} drops to an average performance of 71\%, which should be taken as the most accurate measure of performance for the Helix-predictor.
Again just like for optimisation, an evaluation on set 2, here with a training on set 3, 
yields the best Q\textsubscript{ok}. This supports previous findings that for IMPs with fewer helices the prediction performance increases.
Additionally it seems like that worst performance on the predictor trained on set 1 is cause by the fact that the parameters were optimised on set 2. However as can be seen in Table~\ref{tbl:paramopt} they hardly differ.
What does differ nonetheless though and is likely causing the bad performance are the smaller variety support vectors available.

\begin{table}[!t]
\processtable{\textbf{Performance evaluation of Helix-predictor.} \textbf{Set} denotes the training set of the SVM, the according optimization and final test set are chosen as outlined 
in Table~\ref{tbl:setmatching}. Definitions of the scores can be found in
Table~\ref{tbl:level2scores}. All scores are shown multiplied by 100.\label{tbl:level2results}}
{
%   \begin{tabular}{l|>{$}l<{$} >{$}l<{$} >{$}l<{$} >{$}l<{$} >{$}l<{$} >{$}l<{$} >{$}l<{$} >{$}l<{$}}
  \begin{tabular}{l|cccccccc}
  
\toprule
\textbf{Set} & \textbf{Q\textsubscript{ok}} & $\textbf{Q}_{\textbf{htm}}^{\textbf{\%obs}}$ & $\textbf{Q}_{\textbf{htm}}^{\textbf{\%pred}}$ &\textbf{Q\textsubscript{2}} & 
$\textbf{Q}_{\textbf{2T}}^{\textbf{\%obs}}$ & $\textbf{Q}_{\textbf{2T}}^{\textbf{\%pred}}$
&$\textbf{Q}_{\textbf{2N}}^{\textbf{\%obs}}$& $\textbf{Q}_{\textbf{2N}}^{\textbf{\%pred}}$\\
\midrule
1 & 67 & 93 & 95 & 87 & 76 & 86 & 93 & 87\\
2 & 72 & 94 & 97 & 90 & 78 & 88 & 95 & 91\\
3 & 75 & 94 & 94 & 89 & 80 & 78 & 92 & 93\\
\midrule
Avg & 71 & 93 & 95 & 88 & 78 & 84 & 93 & 90\\
\botrule
\end{tabular}}{}
\end{table}

For secondary structure prediction $\text{Q}_3$ is a common measure. However for the assessment of transmembrane helix predictors, this is not necessarily a good choice, as already mentioned before.
Figure \ref{fig:q2prot} again highlights the comparably little variation in the score with most proteins falling in the range between 90-100\%

% shows the $\text{Q}_2$ distribution for all IMPs. It can be seen that this scores is generally very high, averaging at 88\%. Keeping in mind earlier results this again highlights
% that $\text{Q}_2$ is largely dominated by non-transmembrane residues.
% Most abundant is a $Q_2$ between 90\%-95\% with a frequency of 25.
% no idea why this is an important plot. den kann man im Zweifel toeten, aber Burkhard hatte ihn auch oefter ma

\begin{figure}
 \includegraphics[scale=0.32]{./figures/q2proteins.png}
 \caption{\textbf{Distribution of per protein Q2 scores.} Shown is a histogram of Q2 values calculated for every protein in all three sets of transmembrane proteins.}
 \label{fig:q2prot}
\end{figure}

A comparison of the predicted and observed TMH length distribution shown in Figure~\ref{fig:helixlengths} reveals a substantial correlation. Solely at the ends of the distribution a difference emerges as there are some 
very short but also a few comparably long helices predicted which do not occur within the observed TMHs. This is an important observation to take into account for post-processing of the prediction. There are only few helices of length 1 which
are clearly artefacts and would have to be fixed. However the next higher data is for helices of length four and larger already, closer to the real helix lengths. In total
this suggests that the Helix-predictor already delivers a very meaningful and biologically sound prediction of helices, only slightly underestimating their length. A simple approach of training a third level of SVMs on the output of the Helix-predictor did 
not yield better results and seemed to only reinforce the initial prediction. This did not change either, after adding several descriptive features like a PSSM.

\begin{figure}
 \includegraphics[scale=0.32]{./figures/helixlengthdist.png}
 \caption{\textbf{Helix length distribution.} Displayed is the helix length distribution for all observed helices in the testing sets (green) and also all predicted helices (red).}
 \label{fig:helixlengths}
\end{figure}


\section{Conclusion and outlook}
Overall the performance is satisfying and to some extent even comparable to published methods,  % is that true?.. for memsat it is in some scores but not in the important ones TODO
however only if efficiency for large scale analyses and runtime are not of importance. With the large feature set for the Helix-predictor the assignment is already very time-consuming, but the usage of PredictProtein claims even more runtime.
Therefore the most important future improvement should be directed towards a more thorough feature selection on the one hand and on finding a more efficient and independent way for obtaining these. Additionally this could also introduce
major improvements in prediction performance.

There is many additional work that exceeds the scope of this project but would be vital for a predictor with the objective to compete with the newest generation of TMH predictors. 

A mandatory addition would be the handling of signal peptides as this has long been identified as a problem in the field and subsequently been addressed by newer methods..
After the short cross check for falsely assigned IMPs within the prediction of the 100 soluble proteins it became 
quickly apparent that signal peptides constitute a major source of error that could likely have been avoided with an additional predictor for signal peptides, trained on an additional dataset.

Furthermore re-entrant helices were considered as normal helices during this work, however their unique features make them a viable target for a specialised predictor. Even more so, exclusion of these cases in the 
training set with normal TMHs might lead to an increase in performance due to the decrease in data diversity.

Finally more attention should be put on refinement of the prediction, although clearly not a trivial task.

While not the main point, attention should also be directed towards the SolTMH-predictor which delivers a performance that is not yet on par with current standards although this classification should be much easier to tackle
compared to the complete prediction of helices.
Next, the jury decision based approach should be evaluated with additional testing data to confirm that it has improved performance, compared to the single predictors.

It also remains to be determined whether a evaluation, ignoring unaligned residues in the training data would improve results or how these residues should be treated during training already.

With all these issues resolved the proposed method could be a promising approach, standing out with the unique features from PredictProtein. % ...-_- wasn satz.


%3rd svm%single helices....wie in meinen sol bsp

%evtl confidence norm ansprechen..oder weglassen und hoffen tobi faellts nicht auf
%topology?
%3rd class 0 ? for unknown regions ....


% TODO: %We need to say one contains much less imp than the others. this also explains the results of the sol on this set! (over/underpredicting)... WHERE? this is in results i thought jo: probably to results together with the qok thing

\bibliography{PP1.bib}
\end{document}